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Intranet  Model  and  Metrics 

Measuring  intranet  overall  value  contributions  based  on  a 
corporation  s  critical  business  requirements. 


lthough  many  corporations  store  a  great  deal  of 
information  in  their  corporate  intranets,  few  have 
a  reliable  means  of  measuring  the  effectiveness 
of  their  intranet  portals  to  use  this  information 
to  meet  specific  business  needs.  Turning  infor¬ 
mation  into  knowledge  capital  that  corpora¬ 
tions  can  leverage  quickly  for  competitive 
advantage  requires  a  model  and  metrics 
that  tractably  support  it.  Most  intranet 
portal  measurements  are  based  almost 
exclusively  on  usage  statistics — with  lit¬ 
tle  or  no  thought  given  to  design  or 
user  experience — and  corporations 
apply  them  in  a  non-standardized 
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manner,  providing  no  meaningful  insight  into  how 
well  intranets  help  corporations  achieve  their  strate¬ 
gic  objectives. 

As  IT  professionals  and  business  decision  makers 
seek  ways  to  forge  their  information  into  knowledge 
capital  that  can  be  leveraged  quickly  for  competitive 
advantage,  they  require  a  model  and  supporting  met¬ 
rics  to  do  so.  What  has  been  missing  is  a  comprehen¬ 
sive  model  and  methodology  to  base  measurements 
on  logically  related  groups  of  metrics  which,  when 
measured  periodically,  provide  actionable  steps  to 
optimize  the  efficiency  and  effectiveness  of  intranet 
portals. 


As  IT  professionals  and 
business  decision  makers 
seek  ways  to  forge 
their  information  into 
knowledge  capital  that 
can  be  leveraged  quickly 
for  competitive  advantage 
they  require  a  model  and 
supporting  metrics  to 
do  so. 


Intranet  vs.  Internet 

An  essential  fact  when  measuring  effectiveness  of  cor¬ 
porate  portals  is  to  recognize  and  account  how  their 
purpose  is  similar  to,  but  fundamentally  distinct 
from,  Internet  portals.  Intranets  exist  to  fulfill  differ¬ 
ent  purposes  for  different  constituencies  than  does 
the  Internet.  The  key  difference  lies  in  the  underly¬ 
ing  mission  of  the  portal  itself:  on  the  Internet,  the 
portal  sites’  business  model  is  based  on  attracting  a 
portion  of  the  advertising  budgets  of  corporations 
that  might  otherwise  advertise  in  other  media  (such 
as  print,  TV,  or  radio) .  Thus,  the  general  purpose  of 
the  public  portals  is  to  attract  large  numbers  of  repeat 
visitors,  to  build  online  audiences  with  the  inclina¬ 
tion  to  buy  what  the  portal  advertisers  are  offering. 
These  portals  have  essentially  settled  into  a  one- 
directional  relationship  with  their  users. 

Inside  the  organization,  the  portal  takes  on  an 
entirely  different  character.  It  takes  its  purpose  from 
the  overall  mission  of  the  organization:  to  add  suffi¬ 
cient  value  for  its  customers  to  create  a  sustainable 
business  model.  It  takes  its  features  and  functionality 
from  the  mandate  to  operate  at  world-class  efficiency 
and  effectiveness  in  order  to  remain  competitive. 
Achieving  this  competitiveness  requires  a  bi-direc¬ 
tional  model  that  can  support  the  knowledge  workers’ 
increasingly  sensitive  needs  for  pertinent,  helpful, 
timely  content  and  interactive  information  manage¬ 
ment  tools. 

Today,  a  fundamental  shortcoming  in  trying  to 
increase  value  from  corporate  intranets  is  due  to  a  lack 
of  comprehensive  and  credible  means  by  which  to 
measure  how  effectively  the  portals  meet  the  demands 
of  their  employees  (knowledge  workers)  and  other 
intended  audiences  in  pursuit  of  carrying  out  business 
objectives.  As  Figure  1  illustrates,  most  approaches  to 
metrics  do  not  begin  from  a  strategic  management 
viewpoint  that  takes  into  account  a  prioritization  of 
critical  business  requirements  essential  for  value  cre¬ 
ation.  Far  more  work  in  the  IT  community  focuses  on 
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Figure  1.  Software  engineering 
approach  to  metrics  [2]. 


applying  metrics  to  knowledge 
worker  processes  as  they  impact 
immediate  costs  and  benefits,  and 
this  work  fails  to  recognize  longer- 
term  payback  as  these  processes 
relate  to  company  competitiveness 
(for  example,  processes  that  sustain 
key  business  activities  that  support 
value  creation).  In  order  to  write 
better  software  to  design  intranet 
portals  and  measure  their  perfor¬ 
mance,  value  must  be  understood 
and  linked  to  critical  business 
requirements  with  the  proper  bal¬ 
ance  of  metrics  that  can  be  used  to 
further  derive  better  estimates  of 
return  on  investment  (ROI). 

Few  approaches  to  metrics 
begin  from  a  strategic  manage¬ 
ment  viewpoint,  which  lets  orga¬ 
nizations  prioritize  critical 
business  requirements  essential 
for  value  creation.  Designing  bet¬ 
ter  software  for  intranet  portals 
and  effectively  measuring  the  por¬ 
tals’  performance  requires  linking 
value  to  critical  business  require¬ 
ments  with  the  proper  balance  of 
metrics  that  help  corporations 
derive  more  meaningful  ROI  esti¬ 
mates.  In  effect,  this  focus  would 
close  the  gap  of  understanding 
between  knowledge  workers,  IT 
professionals,  and  business  deci¬ 
sion  makers. 


Strategic  Management 


Economic  Valuation 


\Costs  and  Benefits/ 


Metrics 


Work  to  date  in 
IT  community 


Finance  and 
Strategy 


V 


Software 

Engineering 


Knowledge  Workers 


Front-End  Constituents 

Back-End  Constituents 

•  Accessibility — the  information’s  availability,  reachability, 
and  understandabilty  (that  is,  how  the  information  is 
packaged  and  presented  to  make  it  easier  for  knowledge 
workers  to  understand). 

•  Content  properties — the  characteristics  of  a 
content  item,  such  as  author,  length,  and  name, 
represented  with  a  schema  and  supported  by 
vocabularies  of  metadata. 

•  Communication  of  authoritativeness  and 
importance — proof  of  the  credibility  of  information 
within  an  information  system  that  inspires  confidence 
and  trust. 

•  Domain  information  infrastructure — the  sum  and 
organization  of  all  of  a  corporation’s  data,  taxonomies, 
tools,  and  products.  The  portal  should  include  only 
the  elements  from  these  groups  that  it  can  further 
develop  and  integrate  to  improve  control  of  content 
and  context. 

•  Communication  of  understanding  search — 
demonstration  of  the  information’s  meaning  and 
significance  by  keeping  it  consistent  to  ensure 
acceptance  and  engagement. 

•  Domain  integration  framework — the  virtual 
representation  of  the  relationships  between  a  design’s 
key  elements,  which  shows  how  these  elements 
interact  and  transfer  information. 

•  Information  grouping  and  segmentation — the  logical 
collection  of  relevant  and  similar  information  and  the 
extraction  of  relevant  parts  of  a  document,  respectively. 

•  Information  life  cycle —  the  events  that  recur 
frequently  in  maintaining  the  relevance  and 
accessibility  of  content  in  an  information  system. 

•  Navigation — a  method  of  moving  through  the  domain 
framework  using  visual  presentation  and  consistent 
choices.  Navigation  can  be  local  (vertical)  or  global 
(horizontal). 

•  Search — an  application  that  knowledge  workers  use 
to  find  through  direct  surfacing  or  through  surfacing 
an  obvious  navigational  path. 

•  Personalization — a  method  of  contextualizing 
information  for  a  knowledge  worker  based  on  what  is 
known  about  that  worker. 

•  User  data — the  facts  and  figures  a  knowledge 
worker  maintains  private  access  to  for  knowledge 
retention  and  expansion  (also  referred  to  as 
personalization). 

•  User  assistance — help  available  to  the  knowledge 
worker  while  using  an  information  system,  including 
guidance  on  how  to  use  the  system  or  find  particular 
information. 

Table  1.  Front-  and 
back-end  constituents. 


Intranet  Efficiency  and  Effectiveness  Model 

The  intranet  efficiency  and  effectiveness  model 
(IEEM)  provides  a  needed  framework  for  the  “fam¬ 
ily  of  measures”  approach  by  graphically  depicting 
multiple  indicators  that  derive  the  unique  contribu¬ 
tions  of  IT  at  the  process  level.  It  further  provides 
derivations  for  common  units  of  analysis  (time  and 
money)  by  linking  sets  of  metrics  and  conversion 
ratios  to  business  processes  as  they  relate  to  knowl¬ 
edge  workers,  IT  managers,  and  business  decision 
makers  seeking  to  increase  value. 

Information  technology  does  not  just  contribute 
data;  it  impacts  many  business  processes  that  produce 
results  required  to  sustain  value.  Analyzing  processes 


within  the  IEEM  can  help  corporations  to  better 
understand  which  processes  are  critical  to  sustaining 
productivity.  One  measure  of  intranet  effectiveness  is 
how  well  the  company’s  portals  support  its  business 
requirements.  Research  on  the  IEEM  can  help  iden¬ 
tify  metrics  and  conversion  ratios  that  corporations 
can  apply  to  their  portals  to  determine  where  they 
need  to  focus  efforts  on  meeting  strategic  business 
requirements.  By  dividing  an  intranet  into  segments, 
the  IEEM  helps  corporations  define,  apply,  and  refine 
a  balanced  baseline  of  metrics  for  measuring  what  is 
important,  instead  of  what  is  available. 

To  better  appreciate  the  purpose  of  portals  and  rea¬ 
sons  for  their  occasional  redesign,  it  is  important  to 
understand  the  domains  supporting  information 
management  as  they  relate  to  finding  and  under¬ 
standing  information,  the  domains’  constituents,  and 
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Front  End  Ensure  Institute  Navigation  Quality, 
Authoritativeness,  Personalized  Experience 


Accessibility 
Navigation 
Customization/ 
User  Assistance 
User  Data 


User 

Experience 


g  Sat.  Surveys 
No.  Clicks  to  Find 
Time  to  Locate 


Easy  to  Measure 
Activity  by  Site,  IJS3.5?G 
Time,  and  Duration  ® 


Users 


Back  End 

Develop  Shared  Taxonomy 
Global  Search 
Across  Critical  Content 
Domain  Integration 

Traffic  Volume 

/  \  No.  Hits  per  Link  Frequent  Path 

i  H  ;  Top  10  Downloads  ^n'clue  Users^ 

V  J  Top  1 0  Searches  _ 

Access  Content 

Properties 
Info  Group 
Information 
Life  Cycle 


Comm. 

Authority 

Comm. 

Understand 


Enterprise 


Better  Decisions  (  D 

Reach  1  ** 

More  Creativity  Loya|ty- - AgiNty 

New  Opportunity 

Business  Req. 

-  Harder  to 

Applications '^Measure 
Platform  / 


Domain  Dependencies 
Consistent  Seamless  Interaction  Access 

Domain  Infrastucture 


Portals 


Information 

Architecture 


People,  Process,  Technology 

Challenge  and  Vision:  Given  knowledge  workers  can’t  find  all  information 
needed,  (especially  technical  info  and  experts)  with  portals  that  lack 
consistency  inmanagement  and  few  central  management  services  in  an 
enterprise  intranetthat  is  afflicted  with  duplication/redundancies  and  no 
business  intelligence.develop  an  improved  information  finding  capability  by 
providing  an  intranet  environment  that  consistently  supports  both  browse 
and  search  so  that  knowledge  workers  can  confidently  and  seamlessly  find 
the  technical  information  they  need  to  work  effectively  and  efficiently. 


Information  Grouping 
Navigation 
Personalization 
Search 
Taxonomy 
User  Date 


H  Hard  Metric 


Soft  Metric 


Derived  Metric 


how  segmenting  these  constituents  facili¬ 
tates  highlighting  and  focusing  on  key 
business  requirements.  Three  domains  the 
IEEM  identifies  are: 

•  Front  end,  which  addresses  user-based  factors, 
such  as  accessibility  and  site  navigation; 

•  Back  end,  which  addresses  site-based  factors,  such 
as  personalization,  taxonomy,  and  information 
search;  and 

•  People,  processes,  and  technology,  which 
addresses  knowledge-worker-based  factors,  such 
as  their  vision,  purpose,  and  products  or  ser¬ 
vice — in  short,  how  well  these  factors  support 
business  requirements  that  promote  productivity. 

Constituents  represent  the  data  necessary  in  the 
front-  and  back-end  domains  to  find  information  sup¬ 
porting  the  people,  processes,  and  technology  domain 
(see  the  sidebar  “Requisites  for  Portal  Design”). 

Intranet  Composition 

After  identifying  the  requisites  and  their  con¬ 
stituents,  we  used  a  series  of  diagram  procedures  to 
illustrate  and  analyze  the  composition  of  an 


intranet,  and  used  this  information  to 
determine  the  IEEM  and  its  fundamental 
baseline  of  metrics.  To  determine  associa¬ 
tions  between  metrics  groups  and  critical 
business  requirements,  we  used  an  affinity 
diagram  to  create  a  conceptual  model  separating  the 
intranet  into  distinct  segments  that  underlie  each 
domain:  content,  business  requirements,  design, 
domain  infrastructure,  usage,  and  user  experience. 
This  provided  us  with  greater  resolution  to  map  out 
the  problem  and  solution  space.  On  top  of  this  dia¬ 
gram,  we  identified  the  various  users  and  their  roles 
within  each  segment.  Next,  we  placed  an  interrela¬ 
tionship  diagram  on  top  of  the  affinity  diagram  to 
highlight  pertinent  metrics  (such  as  relevance,  ease 
of  navigation,  and  user  satisfaction  surveys)  and 
their  logical  relationships  between  related  users  and 
their  roles. 

We  further  classified  these  metrics  into  hard,  soft, 
and  derived  forms  [3],  outlined  them  on  the  dia¬ 
grams,  and  put  them  into  a  cause-and-effect  tree  table 
after  consulting  with  a  variety  of  business  decision 
makers  and  knowledge  workers.  Lastly,  using  the 
resulting  IEEM  diagram  and  prioritization  table,  we 
put  all  of  these  factors  into  a  prioritization  matrix  to 


Figure  2.  IEEM  metric 
types  and  examples 
for  all  segments. 
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illustrate  levels  of  importance  and 
to  establish  a  metric  baseline  from 
which  to  begin  measurement  [1], 

The  resulting  theoretical 
model  illustrates  the  placement  of 
the  underlying  and  interdepen¬ 
dent  domains,  segments,  and 
constituents,  as  well  as  the  conse¬ 
quential  and  logical  metric  group¬ 
ings  supporting  business 
requirements.  Figure  2  is  a  simpli¬ 
fied  version  of  this  model.  It 
shows  where  the  segments  fit  into 
the  three  domains.  The  segments 
are  green  to  reflect  the  mix  of 
front-end  (blue)  and  back-end 
(yellow)  factors  and  the  inner  core 
of  the  IEEM  is  offset  in  purple 
(the  darker  shade  of  the  people, 
process,  and  technology  domain 
of  derived  metrics)  to  represent 
the  efficacy  of  relevant  metrics. 

Figure  2  further  separates  effi¬ 
ciency  from  effectiveness.  Effi¬ 
ciency  measures  are  predominantly 
quantifiable  (or  hard)  metrics, 
that  is,  numbers  and  durations  of 
time  or  both.  They  comprise  the 
usage  and  user  experience  seg¬ 
ments.  Effectiveness  measures 
include  efficiency  measures  but 
also  take  into  account  qualitative 
factors  (soft  and  derived  metrics). 

These  measures  make  up  the 
design,  business  requirements,  domain  infrastructure, 
and  content  segments.  Analysts  must  distinguish  and 
appreciate  differences  between  metrics  (hard,  soft,  and 
derived)  in  each  segment  to  attain  the  best  practice 
of  them. 

The  IEEM  includes  an  intranet’s  three  correspond¬ 


ing  audiences:  corporation  business  decision  makers, 
portal  owners  and  managers,  and  users.  Efforts  to  orga¬ 
nize,  prioritize,  and  apply  metrics  for  measuring  an 
intranets  effectiveness  must  consider  all  three  audi¬ 
ences.  For  people  within  these  audiences  to  benefit 
from  enhancements  suggested  by  IEEM  analysis,  busi¬ 
ness  decision  makers  must  know  who  and  where  these 
individuals  are  within  the  corporation  as  well  as  their 
roles  in  fulfilling  business  requirements. 

The  IEEM  focuses  on  strategic  fit,  functional  objec- 


Pri. 

Type 

Metric 

Segment 

Constituent 

Business  Requirement 

1 

S 

Relevance 

Content 

Content 

Properties,  Search 

New  Opportunity,  More  Creativity,  Better 
Decisions,  Value 

2 

H 

Traffic 

Volume 

Usage 

Reach,  ROI,  Loyalty 

3 

H 

Unique 

Users 

Usage 

Reach,  ROI,  Loyalty 

4 

D-»H 

Cross-Portal 

Referral 

Domain 

Infrastructure 

Domain  Information 
Infrastructure 

Reach,  Opportunity,  Creativity,  Disintermediation, 
Value 

5 

S 

Ease  of 
Navigation 

Design 

Navigation,  Comm, 
of  Understanding 

Value,  Loyalty 

6 

H 

Top 

Downloads 

Usage 

Optimization,  Value 

7 

S 

Sat.  Surveys 

User  Experience 

Value,  Loyalty,  Reach,  (Productivity) 

- 

H 

Time  to 
Locate 

User  Experience 

Navigation 

Value,  Loyalty,  (Productivity) 

Must 

S 

Common 

Terminology 

Content,  Domain 
Infrastructure 

Taxonomy 

Value,  Reach 

Must 

D-»H 

Shared 

Topography 

Domain 

Infrastructure 

Domain  Information 
Infrastructure 

Reach,  Opportunity,  Creativity,  Disintermediation 

(Business  Rules  Considered?) 


?  H 

Accessibility 

User  Experience  Accessibility 

Value,  Reach,  More  Creativity,  (Productivity),  New 

Opportunity  (confidential  info,  handled  correctly) 

A  derived  metric  can  become  a  hard  metric  (see  D->  H  above)  as  key  elements  become  concretely  known  and  measured 
with  precision.  Examples  of  this  within  Cross  Portal  References  and  Shared  Topography  above  are  the  ability  to  capture 
all  unique  visitors  and  information  maps  (akin  to  server  topography)  respectively. 

Table  2.  Ranking  of 
key  intranet  metrics 
by  segment. 


REQUISITES  AND  CONSTITUENTS  FOR  PORTAL  DESIGN 

Portals  aim  to  facilitate  knowledge  workers’  discovery  of  the  information  they  need  to  do  their  jobs  better. 

For  information  to  be  easier  to  find  and  useful,  portal  design  must  be  founded  on  the  following  requisites: 

•  Provide  a  reasonable  amount  of  information  and  meta-information; 

•  Reflect  designers’  understanding  of  information  seeking  and  user  behavior; 

•  Group  information  logically  to  facilitate  navigation; 

•  Inspire  confidence  in  the  quality  of  information  and  meta-information;  and 

•  Be  relevant  to  the  knowledge  worker. 

In  addition,  portals  contain  13  constituents  of  discovering  information  that  collectively  sustain  these  requisites.  For  the  sake  of  simplicity  and 
logic,  seven  of  these  constituents  are  grouped  into  the  front-end  domain  and  six  into  the  back-end  domain  in  Table  1.  Because  the  majority  of 
constituents  are  in  the  front-end  domain,  organizations  must  take  front-end  metrics  into  account.  Failing  to  do  so  could  mean  that  their  actions 
to  increase  value  will  have  less  impact.  H 
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WHAT: 

UNIQUE  USERS  (Priority  No.  3) 

Business  Req.  or 
Derived  Metric 

Back  End  or 

WHERE: 

Enterprise 

Hard  Metric 

Front  End  or 

Soft  Metric 

WHO: 

Portal  Owner  and  Portal  Manager  Related 

WHY: 

HOW: 

Best  Bet  or 

Practice 

BUSINESS  PROCESS - ►  < - BEST  PRACTICES - ►  < -  IEEM  Related 


Business 

Issue: 

-  Metric 

Area 

Business 
Question  of 
Web  Site 
Activity 

Business 
Significance 
of  Web  Site 
Report  Solution 

Specific  Metric  Area 

-  Specific  Metric 

Who  Benefits  and  Why 

(User,  Portal,  Enterprise) 

IEEM 

Segment(s) 

IEEM 

Constituents) 

Efficiency  (E) 
or 

Effectiveness  (F) 

Retention: 

-  Loyalty 

-  Value 

How  effectively 
am  1  building 
loyalty  with 
my  visitors? 

Determine  how 
quickly  you  are 
building  your 
user  base  to 
gage  site 
audience 
enlargement  and 
shrinkage  over 
time  and  vis  a 
vis  other  sites. 

Return  Visitor  Rate 

-  Top  Visitors  (authenticated) 

-  Visitors  by  Number  of  Visits 

-  Visitors  Over  Time 

-  Top  Visitors  by  Hits  (Leads) 

-  Top  Visitors  by  Hits  Over  Time 

-  Top  Visitors  by  Leads  Completed 

-  Visits  by  Length  of  Visits 

*  All  of  the  above  for  Returning  Visitors 

-  New  vs.  Returning  Visitors 

User  -  Gains  familiarity  to  site  which  reduces 
overall  frustration. 

Portal  -  Managers  learn  where  return  visitors 
come  from  and  how  many  there  are,  which  a 
partial  indication  that  users  find  the  site  helpful. 

Enterprise  -  When  users  continue  to  return 
to  the  same  site  for  information,  it  indicates 
that  they  find  it  helpful  and  as  well  as  where 
the  sources  of  richness  lie. 

Info  Architecture.: 

-  User  Data 

F 

Optimization: 

-  Loyalty 

-  Value 

-  Agility 

What  do  my 
visitors  come 
back  for? 

Analyze  the 
most  popular 
content  for  my 
return  visitors 
in  order  to 
load  and 
associate  related 
information  to 
meet  demand. 

Return  Visitor  Target  Pages 
(correlate  return  visits  with  content): 

-  Top  Returning  Visitors  by  Hits 

-  Top  Returning  Visitors  by  Hits  Over  Time 

User  -  Popular  downloaded  information 
remains  as  well  as  other  information  like  it 
eventually  being  posted  or  better  associated. 

Portal  -  Managers  learn  which  pages  are 
desired  and  can  load  related  information  to 
meet  demand. 

Enterprise  -  As  portals  more  accurately 

Info  Knowledge 
-  Survey 

Info  Architecture 

F 

-  Optimization 

-  Top  Returning  Visitors  by  Leads 

Completed 

-  Returning  Visitor  Visits  by  Length  of  Visits 

-  Returning  Visitor  Page  Views  Over  Time 

-  Top  Document  and  Content  Group 

-  Info  Grouping 

-  User  Data 

for  Returning  Visitors  Over  Time  monitor  what  information  is  sought  and  add 

additional  information  that  is  related,  the 

Survey  enterprise  is  doing  a  better  job  meeting  the 

-  Internal  Returning  Visitor  Session  expectations  and  business  needs  of  its  users. 

Activity 


tives,  and  the  opportunity  or  necessity  for  making 
process  improvements  as  the  keys  to  success.  In  addi¬ 
tion,  the  IEEM  introduces  a  common  theoretical 
framework  for  measuring  all  facets  of  intranet  processes 
critical  to  assessing  value.  Its  holistic  approach,  how¬ 
ever,  does  not  eliminate  subjectivity  altogether.  The 
IEEM  accounts  for  critical  qualitative  factors  that  other 
commonly  used  measuring  techniques  (which  concen¬ 
trate  on  usage  statistics,  such  as  traffic  volume)  often 
overlook.  In  short,  to  be  more  comprehensive,  the 
model  must  account  for  some  soft  metrics  that  are  nei¬ 
ther  strictly  quantitative  nor  free  of  human  interpreta¬ 
tion  or  assumption.  Nevertheless,  when  parties  agree 
on  a  relatively  well-defined  set  of  performance  metrics, 
a  relatively  unambiguous  collaborative  interpretation  is 
possible  [3]. 

Matrix  for  Metrics  and  Prioritization 

Table  2  shows  an  actual  baseline  estimation  deter¬ 
mined  by  business  decision  makers  and  IT  profes¬ 
sionals  of  the  top  several  metric  areas,  the  segment 
they  come  from,  and  some  of  the  business  require¬ 
ments  they  sustain.  The  prioritization  of  these  met¬ 
rics  is  based  on  the  metric  groups  that  have  the  most 
significant  impact  on  overall  value  from  a  business 
management  perspective.  For  example,  business 
decision  makers  and  IT  workers  might  rate  “rele¬ 
vance”  (a  soft  metric  from  the  “content”  segment)  as 


a  first-priority  metric  because  Figure  3.  ieem  example  of 

of  its  effect  on  new  opportu-  .  .  .  Part  ofone  metric 

r  r  breakdown  for  unique  users. 

nities  for  business,  increased 
creativity,  and  better  deci¬ 
sions.  A  hard  metric,  “traffic  volume”  (from  the 
“usage”  segment)  might  earn  a  priority-two  rating 
for  its  impact  on  reach,  ROI,  and  user  loyalty. 

The  distribution  of  metrics  is  of  additional  interest 
from  a  theoretical  standpoint  because  at  least  one 
metric  area  originates  from  every  segment  (see  the 
“Segment”  column  in  Table  2).  This  supports  our 
assertion  that  to  achieve  accurate  and  comprehensive 
effectiveness  measurements,  analysts  should  take 
more  metrics  from  segments  other  than  the  usage  seg¬ 
ment.  Nevertheless,  it’s  best  to  limit  the  number  of 
critical  metrics  areas  to  include  only  those  that 
directly  correlate  to  a  business  benefit  (although  the 
number  can  vary  for  each  corporation,  the  IEEM 
baseline  example  here  focuses  on  seven.)  Otherwise, 
analysis  can  become  overly  complicated,  threatening 
order,  implementation,  and  credibility. 

Many  corporations  focus  on  routing  metrics,  such 
as  number  of  hits  per  page,  top  10  search  strings, 
most  popular  downloads,  and  number  of  referrals 
from  other  sources  (banner  advertising,  search 
engines,  and  direct  links).  These  metrics  exist  in  far 
greater  number  than  other  metrics  because  they 
address  the  issues  many  organizations  face  today: 
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Web  site  accessibility  and  visibility. 

These  usage-related  metrics  are  also  popular  among 
the  technically  oriented  workers  who  usually  do  the 
measuring  because  they  require  less  time  and  are  more 
mathematically  straightforward  than  the  more  time- 
consuming  soft  metrics  that  measure  user  behavior 
and  experience  in  the  IEEM  front  end.  In  addition, 
routing  metrics  are  relatively  easy  to  understand  at  the 
business  level,  and  the  data  is  relatively  easy  to  collect 
using  Web  server  log  files.  In  fact,  most  Web  analytics 
packages  provide  many  routing  metrics  as  prepack¬ 
aged  reports,  so  deferring  to  these  out-of-the-box  tools 
is  natural.  Unfortunately,  corporations  often  apply 
these  tools  to  their  intranets  as  they  would  to  their 
Internet  Web  sites,  but  there  are  fundamental  differ¬ 
ences  between  the  two  and  simple  substitutions  like 
this  provide  an  inaccurate  assessment  of  an  intranet 
portal’s  performance.  Simple  statistics  on  plumbing 
alone  won’t  provide  business  decision  makers  with  all 
of  the  feedback  necessary  to  track  productivity 
improvements. 

The  audience  most  overlooked  in  ascertaining 
intranet  performance  is  the  user,  even  though  many  of 
the  constituents  necessary  to  sustain  effective  infor¬ 
mation  finding  are  in  the  front-end  domain,  where 
the  knowledge  worker  resides.  Consequently,  when 
selecting  metrics  with  which  to  measure  their 
intranet’s  effectiveness,  corporations  must  carefully 
consider  metrics  from  the  design  and  user  experience 
segments. 

Metrics  and  focused  surveys  inform  analyses  of  user 
behavior  patterns  within  portals  and  help  corpora¬ 
tions  refine  subsequent  metrics  and  surveys.  For 
example,  to  ascertain  why  numerous  visitors  are  aban¬ 
doning  a  particularly  important  site,  a  corporation 
should  examine  user  behavior  through  focused  sur¬ 
veys,  direct  observation,  and  analysis  of  other  metrics 
used  at  the  site. 

A  combined  metrics  and  survey  program  is  also 
critical  to  the  communication  process  and  develop¬ 
ment  of  a  feedback  loop  that  helps  IT  learn  which  ini¬ 
tiatives  provide  the  best  business  value.  For  example, 
raw  visitor  metrics  might  indicate  that  a  corporation 
should  archive  or  discard  an  infrequently  visited 
research  page,  when  in  fact  a  single  recent  access 
might  have  been  the  critical  piece  in  securing  a  major 
new  revenue  stream  for  the  organization.  Surveys  offer 
many  benefits  and  can  lead  to  obvious  but  overlooked 
additions  such  as  providing  an  online  employee  man¬ 
ual  equipped  with  a  search  engine,  reducing  the 
amount  of  time  people  spend  looking  for  the  manual 
and  the  information  in  it. 

To  further  elaborate  on  the  interplay  of  surveys  and 
metrics,  Figure  3  dissects  a  metric  area  from  Table  2 


A  combined  metrics 
and  survey  program  is 
also  critical  to  the 
communication  process 
and  development  of  a 
feedback  loop  that  helps 
IT  learn  which  initiatives 
provide  the  best  value. 


COMMUNICATIONS  OF  THE  ACM  February  2007/Vol.  50,  No.  2 


49 


(unique  users)  to  show  the  reasoning  behind  the 
application  of  metrics  to  IEEM  domains  and  audi¬ 
ences.  The  headings  describe  the  organizations,  peo¬ 
ple,  and  processes  involved.  The  matrix  has  been 
color  coordinated  to  the  IEEM  to  ensure  thorough 
interpretation  of  the  association  between  the  model 
and  metrics,  including  surveys  required.  All  metric 
areas  and  specific  metrics  are  colored  to  better  repre¬ 
sent  what  they  support  and  where  they  belong  in  the 
IEEM  as  they  are  used  with  that  metric  priority  or 
category. 

In  addition  to  being  grounded  in  a  theoretical 
framework,  the  IEEM  methodology  is  practical  in 
that  it  allows  corporations  to  obtain  estimates  using 
many  common  units  (time  being  the  most  useful) 
that  are  directly  traceable  to  specific  pages,  links,  and 
designs  in  a  portal.  Thus,  it’s  possible  to  derive  portal 
effectiveness  in  relatively  practical  ways.  Moreover, 
this  approach  does  not  rely  on  a  particular  software 
program,  so  it  can  run  in  any  network  without  addi¬ 
tional  hardware  or  software  costs,  other  than  the 
server  space  necessary  to  store  log  file  queries. 

Figure  3  is  one  partial  example  of  more  than  a 
dozen  actual  examples  of  baseline  sets  of  metrics  and 
conversion  ratios  resulting  from  high-level  analysis  of 
intranet  efficiency  and  effectiveness.  The  U.S.  Navy, 
the  Defense  Information  Systems  Agency  (DISA), 
and  a  large  Seattle,  WA-based  software  company  are 
studying  and  refining  these  baselines  to  determine  the 
best  techniques  for: 

•  Analyzing  user  behavior; 

•  Using  periodic  soft  metrics  (that  is,  short,  focused 
electronic  surveys)  to  confirm  predominately  hard 
metric  results  related  to  behavior; 

•  Automating  capture  of  optimum  processes  for  sub¬ 
mission  as  business  rules  and  best  practices;  and 

•  Creating  a  single  reporting  service  for  comparing 
portal  performances  consistently  to  gauge  impact 
of  process  changes.1 

Determining  which  complementary  metrics  can  be 
grouped  and  which  groups  best  indicate  how  well  a 
portal  supports  a  business  requirement  can  lead  to 
efficacy  indications.  Refining  these  groupings  from  all 


1Hard  and  soft  metrics  taken  together  with  consideration  given  to  their  strengths  and 
weaknesses  allows  an  enterprise  to  make  informed  decisions  on  the  investment  in,  or 
the  ongoing  value  of  its  data  warehouse  and  portal  system.  Achieving  success  through 
the  use  of  any  performance  metric  will  depend  as  much  on  how  well  it  is  applied  as  it 
does  on  when  it  is  used.  Studies  based  on  samples  and  averages  over  time  can  make  for 
easier  and  more  credible  comparisons.  Hence,  continuous  benchmarking  should  be 
instantiated  to  confirm  and  correct  baseline  measurements  and  conversion  ratios 
through  periodic  (monthly,  quarterly,  annually)  portal  status  reviews  that  measure 
progress  against  previous  baseline  results.  For  these  reviews,  portal  owners  should  use 
the  metrics  to  determine  which  roles  and  content  are  being  underserved  by  the  portal 
and  which  processes  could  correct  this  and  better  leverage  the  portal’s  capabilities. 


intranet  segments  (every  organization  is  unique  and 
therefore  should  work  to  refine  their  own  metrics 
after  periodic  measurements)  helps  corporations 
improve  critical  business  requirements,  such  as  agility, 
disintermediation  (reducing  the  number  of  points 
required  by  process  occurrences),  loyalty,  opportunity, 
and  reach. 

Conclusion 

The  intranet  is  the  most  measurable  medium  ever, 
yet  there  has  not  been  a  successful  demonstration  of 
the  methods  or  strategy  necessary  to  successfully 
implement  a  model  or  measurement  technique  that 
can  indicate  the  effectiveness  of  an  intranet.  To  our 
knowledge,  there  is  little  in  the  literature  that 
addresses  this  matter.  We  have  shown  that  metrics 
can  be  prioritized,  logically  grouped,  and  then  sub¬ 
grouped  with  known  specific,  measurable  hard  and 
soft  metrics  for  each  of  the  three  intranet  audiences 
by  focusing  on  critical  business  requirements  that 
drive  value  and  productivity  within  each.  Since  suc¬ 
cessful  Web  analytics  are  more  a  matter  of  skill  than 
of  technology,  portal  managers  and  business  deci¬ 
sion  makers  might  consider  juxtaposing  their  cur¬ 
rent  approaches  to  intranet  measurement  and  ROI 
on  the  IEEM,  or  using  the  IEEM  as  a  baseline  for 
developing  metrics  that  focus  on  the  critical  business 
requirements  that  derive  competitive  advantage  and 
value.  Understanding  how  seemingly  intangible 
assets  affect  performance  can  mean  the  difference 
between  growth  and  erosion  of  value.  B 
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